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Abstract 

Systems  engineering  and  project  management  are  two  core  engineering  management  processes 
supported  by  core  “ quantitative  ”  disciplines  within  engineering  management  problems.  Traditional 
approaches  to  systems  engineering  focus  on  a  single  system  being  engineered  and  managed  (i.e.,  project 
managed),  while  challenges  addressing  composition  of  systems  of  systems  and  the  reuse  of  systems  for  new 
solutions  require  a  strategic  management  approach  that  promote  a  process  flow  in  which  the  outputs  of 
one  project  (e.g.,  deliverables,  knowledge,  work  documents)  are  captured  for  the  benefit  of  other  projects 
within  and  outside  the  project-based  organization.  Two  other  core  processes  of  engineering  management 
are  therefore  critical  to  be  incorporated  into  this  process  flow:  knowledge  management  and  strategic 
management.  Consequently,  when  applying  complex  simulation  system  or  federations  of  simulation  systems 
for  experimentation,  knowledge  management  and  strategic  management  are  needed. 

The  NATO  Code  of  Best  Practice  (COBP)  for  Command  and  Control  Assessment  is  dealing  with 
similar  challenges.  Within  a  project  in  support  of  PEO  Soldier,  Old  Dominion  University  and  the  United 
States  Military  Academy  developed  new  system  engineering  processes  in  support  of  system  selection  and 
orchestration  that  allow  merging  the  knowledge  and  strategic  management  ideas  with  NATO’s 
recommended  best  practices. 


1  Introduction 

The  study  underlying  this  paper  was  motivated  by  the  need  to  aid  managers  of  projects 
focused  on  developing  complex  simulation  systems  as  well  as  federations  of  systems  to 
cope  with  the  challenges  of  addressing  the  multiple  constraints  set  by  the  customer  and 
sponsor.  In  particular,  the  authors  wanted  to  evaluate  if  it  is  possible  to  use  the  structure 
of  the  NATO  Code  of  Best  Practice  (COBP)  for  Command  and  Control  (C2)  Assessment 
(NATO,  2002)  to  guide  the  use  of  strategic  project  management  and  knowledge 
management  application  in  support  of  the  re-use  of  simulation  resources  beyond  the 
scope  of  a  single  project  or  even  across  the  different  domains  of  the  originally  application 
domain  (procurement,  development,  training,  education,  support  of  operations,  etc.) 
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Litwin,  et  al  (2008)  suggest  a  set  of  key  constraints  in  research  and  development  projects 
focused  on  building  complex  simulation  systems: 

(a)  the  essential  tasks  to  be  used  for  strategic  decision  making  should  be  identified  to 
support  the  selection  or  development  of  relevant  simulation  scenarios; 

(b)  simulation  systems  should  be  selected  based  on  their  ability  to  support  the  evaluation 
of  these  tasks; 

(c)  the  simulated  system  capability  should  be  the  driver  for  the  decision; 

(d)  the  process  should  be  applicable  to  evaluate  alternatives  for  supporting  simulation 
components  and  enable  the  project  manager  to  make  informed  decisions; 

(e)  the  federation  of  these  simulation  systems  should  be  supported  utilizing  the  best 
middleware  available  for  the  task; 

(f)  this  decision  should  be  driven  by  the  functionality  of  the  middleware  and  its  necessity 
in  the  federation  development  process; 

(g)  the  integration  of  systems  and  middleware  should  be  supported  to  the  maximum 
extent; 

(h)  the  decisions  of  model  integrators  should  be  reduced  to  a  minimum,  thus  avoiding 
ambiguity  of  interpretations; 

(i)  existing  solutions  should  be  reused  as  much  as  possible; 

(j)  minimize  the  number  of  supporting  simulation  systems  that  represent  the  scenario; 

(k)  minimize  the  costs  of  obtaining  the  simulation  systems  and  supporting  data; 

(l)  maximize  the  use  of  simulation  system  under  governance  of  the  project  manager; 

(m)  maximize  the  acceptance  of  systems. 

Because  this  set  of  constraints  is  common  in  projects  tasked  to  develop  complex 
simulation  models,  and  because  meeting  these  constraints  will  require  a  considerably 
amount  of  effort  in  deploying  a  systematic  project  management  approach,  the  authors 
believe  that  it  is  critical  to  take  a  strategic  approach  that  promotes  not  only  the  success  of 
a  single  project,  but  of  a  series  of  projects  within  the  project-based  organization. 
Furthermore,  a  strategic  approach  is  always  needed  if  objectives  are  project  overarching, 
as  such  objectives  are  not  necessarily  correlated  with  project  objectives.  In  other  words:  if 
an  objective  is  of  strategic  importance,  it  is  often  counterproductive  to  leave 
accomplishing  this  objective  to  projects  without  additional  guidance  and  incentives.  This 
requires  strategic  project  management. 

In  the  conducted  investigation  we  intended  to  answer  the  research  question:  what  are  the 
tools,  processes,  and  methods  that  support  the  strategic  project  management  of  research 
and  development  projects  tasked  to  build  complex  simulation  systems?  One  of  these 
questions  of  particular  interest  to  the  C2  community  was  to  what  degree  the  framework 
developed  for  the  NATO  COBP  could  be  used  in  this  context,  as  the  COBP  already 
addresses  re-use  of  study  results  in  related  and  relevant  domains. 

In  the  next  sections,  we  provide  fundamental  background  on  the  concepts  of  knowledge 
management  and  strategic  project  management  that  we  applied  in  the  development  of  the 
answer  of  our  research  question,  followed  by  a  proposal  of  a  comprehensive  systems 
engineering  approach  that  is  based  on  the  latest  developments  in  the  field  of  project 
management. 
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2  Knowledge  Management  and  Strategic  Project  Management 

Knowledge  management  aims  to  address  the  challenges  faced  by  modem  organizations 
of  competing  and  improving  performance  through  knowledge  (Davies,  2000). 
Knowledge  management  is  intended  to  use,  improve,  maintain,  and  create  organizational 
capabilities  to  generate  sustained  competitive  advantage  in  organizations  (Yeung,  et  al, 
1999).  Knowledge  management  is  commonly  defined  as  the  processes,  tools,  and 
techniques  that  make  available  the  right  knowledge  to  the  right  knowledge  worker,  at  the 
right  time.  Another  definition  of  knowledge  management  comes  from  Luthans,  he 
redefines  Knowledge  Management  as  “the  development  of  tools,  processes,  systems, 
structures,  and  cultures  explicitly  to  improve  the  creation,  sharing,  and  use  of  knowledge 
critical  for  decision  making.  ”  Knowledge  management  in  support  of  project  management 
is  a  relativity  new  discipline.  Researchers  are  currently  developing  and  improving 
methods  and  tools  to  acquire,  store,  and  disseminate  information  and  tangible  and 
intangible  knowledge  throughout  project-based  organizations  (Love,  et  al,  2003). 

This  is  valid  in  the  military  context  as  well.  In  particular  the  advancement  of  the  Internet 
supported  that  all  of  the  military  services  are  currently  developing  and  applying  web 
portals  as  single  point  access.  Examples  are  the  Army  Knowledge  Online  and  the  Navy 
Knowledge  Online  portals  the  authors  are  familiar  with.  These  portals  are  used  to  train 
soldiers  on  their  duties,  prepare  them  for  deployments,  etc.  They  are  also  used  for 
transferring  a  working  knowledge  of  an  individual  person,  their  experiences,  and 
education  to  another  person  whom  may  be  replacing  them  within  an  organization.  It  is 
also  needed  to  align  projects,  introduce  project  overarching  standards  and  procedures, 
and  support  the  transfer  of  project  results  for  reuse.  Although  knowledge  management 
initiatives  have  been  lately  supported  by  the  need  to  address  the  knowledge  loss  that  faces 
organizations  due  to  the  retirement  of  their  baby  boomers,  knowledge  management  is 
more  than  that,  it  is  important  to  the  immediate  organization  (project  organization)  to 
avoid  re-inventing  the  wheel  and  have  to  solve  the  same  problem  again  and  again.  For  the 
overarching  organization  (project-based  organization)  it  provides  the  opportunity  to  build 
a  social  capital  that  will  support  the  intellectual  capital  and  consequently  lead  to 
enhanced  performance,  capabilities,  and  competitiveness  (Landaeta,  2008). 

Strategic  project  management  has  been  a  term  lately  used  by  researchers  and  practitioners 
of  project  and  program  management  to  refer  to  the  effective  and  efficient  management  of 
project-based  organizations.  Strategic  project  management  focuses  on  the  providing 
means  to  the  management  enabling  them  to  provide  guidance  for  more  than  one  project 
at  a  time  in  a  way  that  the  strategic  intent  of  the  project-based  organization  is  met 
(Callahan  &  Brooks,  2004;  Green,  2005;  Grundy  and  Brown,  2002).  Strategic  project 
management  focuses  on  the  best  utilization  and  alignment  of  the  resources  of  the  project- 
based  organization  to  meet  its  vision  and  goals. 

Knowledge  and  information  have  been  suggested  as  being  resources  of  projects 
(PMBOK,  2002).  Therefore,  they  are  one  of  the  key  aspects  of  strategic  project 
management  because  projects  rely  on  information  and  knowledge  (feedback)  from 
previous  projects  executed  within  the  project-based  organization.  Information  and 
knowledge  are  traditionally  captured  in  documents  like  project  plans,  project  journals, 
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lessons  learned,  best  practices,  etc.  They  all  contribute  to  the  knowledge  base  of  a 
project-based  organization  and  therefore  enable  the  effective  and  efficient  utilization  of 
the  organization  body  of  knowledge  across  projects  (i.e.,  the  strategic  management  of 
projects’  knowledge).  Project  management  and  technical  implementation  success  of  a 
single  project  hinges  on  past  experiences  and  the  transfer  of  knowledge  from  preceding 
projects. 

Knowledge  management  and  strategic  project  management  are  taught  within  the  Armed 
Forces  in  project  management  courses  and  in  leader  education.  However,  their 
applicability  for  the  management  of  “virtual  systems,”  such  as  the  simulation  systems 
used  for  training  and  education,  analysis,  and  experimentation  (and  in  the  future 
hopefully  increasingly  for  support  of  operations  as  well)  has  not  been  the  focus  of 
published  research.  The  studies  underlying  this  paper  were  targeting  to  close  this  gap. 
The  hypothesis  to  test  was  that  “it  is  possible  to  significantly  improve  the  reusability  of 
simulation  resources  in  cross  domains  if  the  recommendations  of  knowledge  management 
and  strategic  project  management  are  applied  in  addition  to  traditional  M&S  project 
management.  ” 

3  The  NATO  Code  of  Best  Practice  for  C2  Assessment 

The  NATO  COBP  for  C2  Assessment  is  rooted  in  operations  research  methods.  It 
recommends  best  practices  for  the  structure  of  C2  evaluation  projects.  Since  2007  the 
COBP  has  been  adapted  as  a  standard  within  the  Joint  Staff  and  Office  of  the  Secretary  of 
Defense  (OSD)  Networks  and  Information  Integration  (Nil). 

Figure  1  shows  the  structure  of  the  COBP  processes  and  their  main  domains. 


\  / 

Assess 

Risk 

|  Products  K 

i  i 

L _ 

*  _ 

Figure  1:  Overview  NATO  COBP 

Within  the  first  phase  of  a  C2  related  operations  research  study,  the  problem  formulation 
and  the  proposed  solution  strategy  is  conducted  based  on  the  sponsor’s  problem. 
Understanding  the  problem  on  the  researcher’s  side  and  how  it  is  going  to  be  solved  on 
the  customer’s  side  are  the  main  points.  In  the  second  phase,  the  scope  and  constraints  for 
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the  study  are  captured:  what  are  the  main  issues  to  address,  what  scenarios  to  use  for  the 
evaluation,  and  what  metrics  can  be  used  to  measure  success.  The  third  phase  conducts 
the  evaluation  using  tools  (one  of  them  can  be  simulation)  and  using  and  producing  data. 
In  the  final  phase,  the  risk  is  assessed  and  the  products  for  the  sponsor  are  prepared. 

Since  the  COBP  deals  with  similar  challenges  as  explained  in  the  introduction,  we 
wanted  to  evaluate  in  particular  to  supporting  knowledge  management  and  strategic 
project  management  can  be  guided  by  the  processes  recommended  by  the  COBP.  In  other 
words,  the  hypothesis  was:  “It  is  possible  to  use  the  structure  of  the  COBP  to  identify 
phases  and  required  support  that  can  be  provided  knowledge  management  and  strategic 
project  management  enabling  cross-domain  re-use  of  simulation  resources.  ” 

4  Proposal  for  a  Systems  Engineering  Approach 

While  evaluating  on  the  one  hand  side  traditional  project  management  methodologies  and 
artifacts  -  e.g.,  the  statement  of  work  (SOW),  work  breakdown  structure  (WBS),  resource 
management  -  and  on  the  other  hand’s  side  NATO’s  recommended  COBP  phases,  three 
core  phases  were  recognized  and  linked  to  develop  a  reusable  project  specific  strategic 
project  management  process  (SPMP).  This  process  opens  the  way  for  reusability  of  the 
simulation  resources  of  a  single  project  for  re-use  by  other  projects  and  as  such  should 
enable  the  project-based  organization  to  become  more  effective  and  efficient  over  time. 

captures  these  identified  core  phases  to  illustrate  a  SPMP  flow  that  can  be  utilized  for 
modeling  and  simulation  (M&S)  federation  development  or  the  integration  of  simulation 
resources  into  other  domains.  The  three  core  phases  are  initial  planning,  refining,  and 
implementation.  Each  phase  requires  supporting  documentation  and  sub-processes  to 
deliver  a  product: 


Table  1.  SPMP  Core  Phases  and  Outputs 


Initial  Planning 

Project  Proposal 

Refining 

Project  Work  Plan 

Implementation 

Final  Product 

Supporting  documentation  noted  in  Table  1  contains  key  information  and  necessary 
knowledge  required  to  take  an  M&S  conception  to  reality  -  i.e.  development  cycle.  It 
should  be  noted  that  each  phase  has  sub-processes  where  several  reiterations  may  be 
needed  to  facilitate  proper  coverage  of  the  requirements  and  to  apply  lessons  learned,  best 
practices,  and  near-miss  events  from  previous  projects  i.e.  knowledge  management. 

Project  management  and  the  technical  implementation  success  of  a  single  project  hinges 
on  past  experiences  and  the  transfer  of  knowledge  from  preceding  projects.  Different 
artifacts  are  identified  to  promote  this  continuous  improvement  cycle:  study  plan,  project 
journal,  and  risk  management.  A  study  plan  and/or  project  journal  maintained, 
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communicated,  and  socialized  to  all  members  promotes  a  well  informed  team.  Risk 
analysis  and  management  is  recommended  in  order  to  avoid  unforeseeable  difficulties 
and/or  setbacks  and  is  enabled  by  using  knowledge  collected  from  previous  projects  to 
assess  and  address  the  consequences  of  reducible  and  irreducible  uncertainty. 

Standardizing  the  project  management  products  in  order  to  support  their 
understandability,  transparencies,  and  reusability  is  a  necessity  to  enable  a  strategic 
project  management  approach  as  recommended  in  this  paper.  If  introduced  correctly,  no 
additional  work  is  required  within  the  project,  as  the  project  management  products  are 
needed  within  the  project  in  any  event.  However,  by  doing  them  in  a  standardized  way 
increases  the  reusability  and  sharing  across  the  boundaries  of  a  project  and  of  the  project- 
based  organization,  but  this  latter  topic  lies  outside  the  boundaries  of  this  paper. 

The  resulting  requirements  enabling  an  overarching  integrative  approach  assume  that 
M&S  services  need  to  be  accessible  via  a  knowledge  repository  in  which  they  are 
described  in  a  standardized  way.  A  recommended  systems  engineering  approach,  the 
Model  Driven  Architecture  TM  (MDA),  aid  the  management  of  knowledge  in  this  type  of 
projects.  Here  requirements  are  used  to  formally  capture  M&S  models  in  descriptive 
artifacts  to  realize  captured  knowledge  of  used  components  and  how  they  are  contributing 
to  the  process  builds  the  knowledge  repository  with  valuable  information.  However,  it 
should  be  pointed  out  that  this  does  not  imply  that  a  technical  MDA  approach  is 
mandatory  as  well. 

The  proposed  SPMP  illustrated  in  Figure  2  may  be  utilized  as  a  management  tool  for 
M&S  federation  development;  however  it  can  be  adapted  for  any  project  requiring 
dynamic  development  of  software  or  simulation  models.  It  contains  three  core  phases: 
Initial  Planning,  Refining,  and  Implementation  with  each  providing  a  different  product: 
Project  Proposal,  Project  Work  Plan,  and  Final  Product  respectively. 

From  Figure  2 

it  can  be  noted  that  there  are  two  domains  which  the  SPMP  functions  in  -  the  overall 
domain,  Project-Based  Organization,  and  the  inner  domain,  the  Project.  These  boundaries 
define  where  individual  projects  are  part  of  a  collective  project-based  organization.  It  is 
imperative  to  utilize  a  reusable  Knowledge  Repository  during  a  development  cycle  in 
order  to  capitalize  on  previous  project  outcomes.  Queries  may  be  called  to  retrieve  data 
and  submissions  should  be  made  to  a  repository  to  ensure  feedback  (e.g.  lessons  learned, 
best  practices,  near-miss  events,  and  supporting  documentation)  within  the  project.  This 
practice  allows  overarching  project-based  organizations  to  expand  critical  knowledge  and 
become  more  efficient  over  time. 

For  a  project  to  be  initiated,  a  sponsor  must  present  a  problem  by  explicitly  annotating  the 
strategic  importance  of  the  problem  to  be  solved.  Only  then,  upon  completion  of  the 
strategic  importance  documentation,  the  SPMP  can  commence.  The  first  phase  of  the 
SPMP  -  Initial  Planning  -  contains  two  sub-processes:  Problem  Formulation  (What?)  and 
Solution  Strategy  (How?).  These  two  sub-processes  will  aid  in  developing  and  refining 
supporting  documentation  to  evaluate  what  the  system  will  do  in  support  of  which 
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missions,  Table  2,  ultimately  leading  to  a  Project  Proposal.  It  should  be  noted  that  several 
reiterations  may  be  needed  to  resolve  any  outstanding  issues  and  to  close  the  gaps  of  any 
missing  requirements  in  order  to  provide  a  robust  and  complete  project  proposal. 


Figure  2:  Strategic  Project  Management  Process  (SPMP)  for  the  Research  and  Development 
Projects  Focused  on  Building  Complex  Simulations 
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Table  2.  Initial  Planning  Supporting  Documentation 


Supporting  Documentation 


Statement  of  Work  (SOW) 

Summary  of  Technical  Specifications 

Contractual  Constraints 

Assumptions 

Stakeholder  Analysis 

Responsibility  Matrix 

Work  Breakdown  Structure  (WBS) 

Resource  Analysis 

Cost  Analysis 

Study  Plan 


The  second  phase  of  SPMP  -  Refining  -  commences  after  a  contract  has  been  awarded 
and  refining  spatial  and  contractual  elements  is  required.  There  are  three  sub-processes: 
Indentify  Human  and  Organizational  Factors  (evaluating  where  they  are  now  and  how 
they  operate),  Contextualize  Human  and  Organizational  Factors  (placing  them  into  the 
overall  scenario),  and  Select  Measures  of  Merit  (MoM)  (identifying  the  important 
concepts  and  processes,  their  role,  and  how  to  measure  success  or  failure).  These  develop 
additional  supporting  documentation,  as  shown  in  Table  3Error!  Reference  source  not 
found.,  and  ultimately  deliver  a  Project  Work  Plan.  During  the  contextualization  sub¬ 
process  several  supporting  documents  (e.g.  responsibility  matrix,  resource  analysis)  from 
the  previous  phase  must  be  revised  to  add/change  information  pertaining  to  the  awarded 
contractual  constraints. 

Table  3.  Refining  Supporting  Documentation 


Supporting  Documentation 


Critical  Knowledge 
Key  Communications 
Uncertainty  Analysis 
Metrics 


To  produce  results,  MoM  is  undoubtedly  required.  The  need  for  MoM  -  explicit  or  not  - 
is  to  classify  a  result  (e.g.  states,  events)  by  the  assignment  of  success  or  failure. 
Applying  formal  MoMs,  provide  a  better  understanding  of  the  project  and  allow  the 
results  to  be  compared  against  each  other  and  with  other  projects  within  the  project-based 
organization.  Also,  the  use  of  formal  standardized  MoMs,  provides  the  knowledge 
repository  with  feedback  and  input  for  future  projects. 
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The  final  phase  of  SPMP  -  Implementation  -  starts  immediately  after  the  project  work 
plan  is  complete  and  the  project  development  work  begins.  There  are  two  sub-processes: 
Select  Methods  and  Tools  and  Data  Selection  which  develop  additional  supporting 
documentation  for  the  project,  as  shown  in  Table  4Table  ,  and  delivers  a  product  to  the 
sponsor  for  review.  Again  several  iterations  of  the  sub-processes  may  be  required  to 
rectify  any  missing  data  requirements  identified  in  the  data  selection.  It  should  be  noted 
that  during  the  selection  of  methods  and  tools  sub-process,  revisits  may  be  required  to  the 
solution  strategy  sub-process  to  update  previous  supporting  documentation  (e.g.  WBS, 
resource  analysis,  study  plan)  to  maintain  a  consistent  and  complete  SPMP  solution. 

Table  4.  Implementation  Supporting  Documentation 


Supporting  Documentation 


Simulation  Selection 
Systems  Engineering  Methodology 
Development  &  Deployment 
Data  Documentation 


Before  a  product  is  released  to  the  sponsor,  a  final  Product  Review  is  required.  During 
the  review,  the  product  is  tested  and  compared  to  the  previously  selected  MoMs  and 
requirements  set  forth  by  the  sponsor.  During  the  review,  key  information  about  the 
project  should  be  filed  in  supporting  documentation,  as  shown  in  Table  5Table.  Beside 
the  supporting  documentation  other  key  information  (e.g.  major  findings, 
recommendations)  should  be  annotated  and  communicated  to  the  teams  and  the  sponsor. 

Table  5.  Product  Review  Supporting  Documentation 


Supporting  Documentation 

Shortfalls 
Lessons  Learned 


Risk  and  Uncertainty  Analysis  is  an  ongoing  progression  as  a  project  moves  through  the 
SPMP.  Every  phase  of  the  SPMP  introduces  new  forms  risk  and  uncertainty  pertaining  to 
the  project.  A  project  manager  needs  to  take  all  aspects  of  risk  into  consideration  and 
document  the  process  along  the  way.  Proper  analysis  of  risk  and  uncertainty  provides 
invaluable  supporting  documentation  to  top-level  management  and  all  stakeholders.  If  the 
appropriate  documentation  is  provided,  risk  and  uncertainty  management  deliver  valuable 
support  enabling  the  project  team  to  deliver  their  product  on  time  and  as  described  by  the 
specifications. 

There  are  two  critical  points  -  called  External  Reviews  -  within  the  SPMP  flow  where 
the  teams/stakeholders  should  review  all  available  supporting  documentation  and/or 
products.  These  reviews  should  occur  between  the  initial  planning  and  refining  phase  and 
between  the  implementation  and  product  review  phase.  External  review  promotes  team 
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cohesiveness,  cross-functional  communication,  and  provides  an  avenue  for  decision 
makers  to  become  informed  about  the  process  and  project  status. 

As  Figure  2  depicts  several  transition  points  to  previous  processes,  it  should  be  noted  that 
depending  on  the  project,  process,  PM,  etc.  additional  decision  points  may  be  required  to 
“revisit”  previous  stages  or  to  refine  supporting  documentation.  For  example,  if  a  project 
is  midway  through  its  development  cycle  and  the  PM  determines  a  change  to  the  proposal 
and/or  contract  is  warranted  -  due  to  technical  limitations  -  a  review  of  and  changes  to 
supporting  documentation  may  be  required.  Therefore,  the  labeling  of  Initial  Planning 
Phase  (Pre-Contract)  and  Refining  Phase  (Awarded  Contract)  may  be  deceiving  to  the 
reader. 

The  final  two  elements  of  the  SPMP  that  are  noteworthy  are  the  Project  Journal  and 
Study  Plan.  A  project  journal  is  a  chronological  continuous  document  of  key  events 
containing  information  (e.g.  meeting  time  and  location,  who  attended,  agenda,  what  was 
accomplished,  what  was  outstanding,  lessons  learned  by  those  who  attended).  The  project 
journal  should  commence  with  the  first  event  in  the  first  phase  -  initial  planning.  A  study 
plan  is  considered  a  “playbook.”  It  contains  problem  formulation  and  solution  strategy 
plans  for  all  stakeholders  and  especially  for  the  PM  (NATO,  2002).  This  study  plan  is  a 
management  tool  which  provides  detailed  guidance  with  a  time  phased  execution  plan 
linking  all  of  the  supporting  documentation  (e.g.  SOW,  WBS,  etc.)  together  promoting  a 
smooth  flow  for  the  solution  strategy. 

The  above  recommended  SPMP  provides  a  reusable  project  specific  process  flow  for 
PMs  to  develop  an  intelligent  strategy  and  a  “plan-of-attack”  to  solve  complex  M&S 
federation  problems  based  on  past  experiences.  This  allows  project-based  organizations 
to  become  more  effective  and  efficient  over  time  and  expand  their  critical  knowledge. 

5  Summary  and  Recommendations 

Both  hypotheses  were  validated  within  the  study,  although  the  validation  is  limited  to  the 
results  within  the  study.  The  findings,  however,  were  applied  in  support  of  the  Program 
Executive  Office  (PEO)  soldier  under  realistic  constraints,  as  the  team  conducted 
experiments  and  research  in  support  of  a  real  study.  Based  on  our  observation,  several 
recommendations  can  be  justified: 

Several  project’s  knowledge  management  tools  and  methods  that  support  the  strategic 
management  of  projects  were  identified  during  the  investigation.  We  recommend  senior 
managers  and  program  and  project  managers  of  organizations  focused  on  performing 
research  and  development  projects  tasked  to  build  complex  simulation  systems  to 
evaluate  the  implementation  of  Project  Journals,  Lessons  Learned  Processes,  and  a 
Knowledge  Repository: 

•  Project  Journal:  Loo  (2002)  conducted  a  study  on  project  journaling  and  proved 
it  to  be  beneficial  for  project  management.  He  concluded  that  “journaling  can 
facilitate  learning  specific  skills  including  interpersonal  communication,  conflict 
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management,  managing  effective  meetings,  managing  stress,  and  leadership 
skills.”  This  learning  is  accomplished  by  “reflecting”  thoughts  in  a  project 
journal.  Reflecting  is  carried  out  in  three  stages: 

(1)  self  awareness  of  the  stimulated  learning  situation  (positive  or  negative  and/or 
uncomfortable  circumstances), 

(2)  self  criticizing  of  the  situation,  and 

(3)  self  development  of  new  perspectives  based  on  the  above  discoveries. 

Project  Journaling  is  an  excellent  learning  tool  for  all  personnel  within  a  project 
and/or  project-based  organization.  Journaling  may  be  used  by  organizations  to 
build  teams,  improve  management  skills,  and  improve  organizations  overall  (Loo, 
2002). 

Project  Journals  can  be  extended  beyond  personal  use  -  they  can  be  used  to 
capture  important  events  (e.g.  bi-weekly  meetings,  technical  reviews,  and 
brainstorming  sessions)  that  take  place  during  a  project’s  development  cycle. 
Possible  elements  to  capture  within  a  project  journal,  combined  with  the  above 
mentioned,  should  allow  project  and  project-based  organizations  to  study:  how 
events  occur,  how  the  events  were  performed,  why  were  the  events  performed  in 
that  way,  etc.  The  knowledge  gained  during  this  process  should  then  be  captured 
in  a  knowledge  repository  and  transferred  across  projects  within  a  project-based 
organization.  As  Loo  discovered  with  individuals,  applying  project  journaling  to 
projects  should  allow  project-based  organizations  to  improve  their  performance 
by  reflecting  on  past  experiences  and  applying  those  experiences  to  future 
projects.  However,  as  this  paper  is  based  on  one  supported  project,  the  amount  of 
re-use  could  not  be  measured  as  a  success  factor  for  our  activities. 

Lessons  Learned  (L/L):  U.S.  Army  defines  Lessons  Learned  (L/L)  as 
“knowledge  or  an  understanding  gained  by  experience  either  by  a  negative  or 
positive  experience  (Center  for  Army  Lessons  Learned,  2008).  Before  capturing 
L/L,  it  should  be  determined  the  L/L  are  noteworthy,  valid,  and  relevant  to  a 
particular  subject  before  they  are  officially  documented.  L/L  are  developed  at  the 
macro  level  (management)  and  the  micro  level  (technical),  each  having  their 
scope  of  detail  (finer  details  of  implementation  go  into  technical  L/L).  L/L  may 
enclose  or  address  topics  of  interest,  provide  information  of  an  event,  etc.  and  not 
restate  doctrine  or  policies.  Examples  of  L/L  are:  what  should  have  been 
available,  what  was  available,  what  current  solutions  could  have  helped  to  close 
the  gap,  etc. 

During  PEO  Soldier’s  task  it  was  determined  that  the  technical  teams  should  have 
used  Platform  Independent  Models  (PIM)  during  the  planning  phase  instead  of 
theoretical  Unified  Modeling  Language  (UML)  diagrams.  The  diagrams  proved  to 
be  too  voluminous  for  real-world  use  and  caused  delays.  This  example  is  an 
excellent  point  to  be  made  for  follow-on  projects.  For  strategic  project 
management,  project-based  organizations  must  have  a  plan  in  place  to  capitalize 
on  the  full  potential  of  L/L  -  they  must  accumulate,  validate,  store,  disseminate, 
and  reclaim  L/L  to  achieve  organizational  goals  and  objectives.  Accumulation  of 
L/L  should  be  garnered  from  sources  internal  and  external  to  the  project 
organization  and  contain  positive  and  negative  experiences.  Established 
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guidelines  and  standards  permit  a  streamline  process  for  validating  L/L.  During 
the  validation  process,  key  personnel  and  subject  matter  experts  (SME)  review 
and  tailor  the  L/L  preparing  them  for  storage  in  the  repository.  Captured  L/L  may 
be  applied  to  a  project-based  organization  via  a  knowledge  repository.  However, 
some  standardization  is  required  to  ensure  proper  data  matching  is  maintained 
within  the  repository  and  the  organization.  Without  standards  of  validating, 
storing,  and  reclaiming  those  L/L  cannot  be  effective.  Project-based  organizations 
are  responsible  for  organizing  L/L  and  developing  plans  to  disseminate  and 
applying  L/L  to  follow-on  and/or  recurring  projects.  New  projects  benefit  from 
previous  experiences  by  reducing  the  “learning”  curve  and  ultimately  reducing  the 
development  time  of  a  project  -  especially  those  within  the  same  project-based 
organization.  The  L/L  process  is  an  ongoing  process  starting  from  the  beginning 
of  a  project  and  should  not  be  left  until  the  end  of  a  project  to  begin  capturing 
L/L.  Every  noteworthy  event  should  be  documented,  organized,  and  stored  for 
future  use  enabling  project-based  organizations  to  improve  over  time.  Within  our 
project,  we  applied  these  principles  but  did  not  define  explicit  metrics  to  capture 
the  improvements.  Such  metrics  are  topic  of  ongoing  research. 

•  Knowledge  Repository:  The  resulting  requirements  enabling  an  overarching 
integrative  approach  assume  that  M&S  services  need  to  be  accessible  via  a 
knowledge  repository  in  which  they  are  described  in  a  standardized  way  -  these 
ideas  are  based  on  Model  Driven  Architecture  ™  (MDA)  (  Object  Management 
Group  (OMG),  2007).  Requirements  to  formally  capture  M&S  models  in 
descriptive  artifacts  realize  the  captured  knowledge  of  used  components  and  how 
they  contributed  to  the  process.  This  builds  the  knowledge  repository  with 
valuable  information. 

However,  it  should  be  pointed  out  that  this  does  not  imply  that  a  technical  MDA 
approach  is  mandatory  as  well.  Sinclair  suggested  a  common  data  infrastructure, 
Error!  Reference  source  not  found.,  which  allows  organizations  to  “reuse” 
knowledge  within  a  project-based  organization  by  utilizing  a  repository  (Tolk  and 
Sinclair,  2002).  This  infrastructure  is  not  exclusive  and  is  considered  a  static 
model.  However,  for  MDA  methodologies  to  be  applied  to  this  framework,  a 
model  needs  to  be  modified  and  adapted  to  allow  dynamic  content.  For  example, 
not  all  M&S  federation  projects  are  the  same  (e.g.  different  requirements,  ideas, 
data)  and  need  to  be  handled  in  different  ways.  Therefore,  the  common  data 
infrastructure  may  need  to  be  modified  to  accommodate  these  differences. 

Tolk  and  Sinclair  (2002)  used  the  COBP  to  motivate  the  use  of  data  repositories  to  enable 
re-use  of  operational  research  study  results  in  other  studies.  Figure  3  summarizes  the 
recommended  approach.  It  can  be  seen  that  the  Data  Available  element  is  correlated  to 
the  knowledge  repository  and  the  Study  Data  element  is  information  from  the  current 
project  -  therefore  transferring  knowledge  from  the  repository  and  applying  it  to  the 
current  project.  If  this  structure  is  extended  respectively,  it  does  not  only  serve  the 
sharing  of  data  to  be  used,  but  all  study  results  and  artifacts. 
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Figure  3:  Common  Data  Infrastructure  for  Reuse  of  Knowledge 

Finally,  as  the  community  is  currently  moving  towards  net-centric  organizations,  services 
that  are  offered  to  the  community  have  to  be  described  allowing  their  identification, 
selection,  and  orchestration.  This  descriptions  need  to  annotate  the  services  as  metadata 
and  describe  services,  the  necessary  inputs,  the  resulting  outputs,  and  how  to  access  these 
services.  This  description  is  posted  to  a  repository.  Service  consumers  go  to  this 
repository  and  search  for  potential  solutions  to  their  problem.  If  they  find  a  fit  based  on 
the  description,  they  use  the  mechanisms  described  to  access  the  service.  The  more 
agreement  we  achieve  regarding  the  artifacts  we  use  to  describe  the  systems  and  services, 
the  easier  this  communication  will  be. 

The  recommended  Strategic  Project  Management  Process  (SPMP)  shown  in  Figure  2  and 
document  in  section  4  of  this  paper  merges  the  ideas  of  strategic  project  management, 
knowledge  management,  and  the  NATO  COBP  enabling  cross-domain  re-use  of 
simulation  resources.  This  shows  a  new  and  successful  application  of  the  ideas  captured 
in  the  NATO  COBP  and  its  broad  applicability  in  support  of  warfighter  challenges. 
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Structure  of  the  Presentation 
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Why  improve  the  management  of  experimentation? 

INTRODUCTION 
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Motivation  for  the  Study 

•  Need  to  aid  managers  of  projects 

-  Focused  on  developing  complex  simulation  systems 

-  Applicable  to  federations  of  systems 

to  cope  with  the  challenges  of 

-  addressing  the  multiple  constraints  set 

-  by  the  customer  and  sponsor. 

•  Evaluate  usability  the  structure  of  the  NATO  Code  of  Best  Practice  (COBP) 
for  C2Assessment 

-  to  guide  the  use  of  strategic  project  management  and  knowledge 
management  application 

-  in  support  of  the  re-use  of  simulation  resources  beyond  the  scope  of  a 
single  project 

-  across  the  different  domains  of  the  originally  application  domain,  such 
as  procurement,  development,  training,  education,  support  of  operations 
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Key  constraints  in  R&D  projects 

•  R&D  projects  with  focus  on  building  complex  simulation  systems: 

-  the  essential  tasks  to  be  used  for  strategic  decision  making  should  be 
identified  to  support  the  selection  or  development  of  relevant  simulation 
scenarios; 

-  simulation  systems  should  be  selected  based  on  their  ability  to  support 
the  evaluation  of  these  tasks; 

-  the  simulated  system  capability  should  be  the  driver  for  the  decision; 

-  the  process  should  be  applicable  to  evaluate  alternatives  for  supporting 
simulation  components  and  enable  the  project  manager  to  make 
informed  decisions; 

-  the  federation  of  these  simulation  systems  should  be  supported  utilizing 
the  best  middleware  available  for  the  task; 

-  this  decision  should  be  driven  by  the  functionality  of  the  middleware  and 
its  necessity  in  the  federation  development  process; 
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Key  constraints  in  R&D  projects  (cont.) 


-  the  integration  of  systems  and  middleware  should  be  supported  to  the 
maximum  extent; 

-  the  decisions  of  model  integrators  should  be  reduced  to  a  minimum, 
thus  avoiding  ambiguity  of  interpretations; 

-  existing  solutions  should  be  reused  as  much  as  possible; 

-  minimize  the  number  of  supporting  simulation  systems  that  represent 
the  scenario; 

-  minimize  the  costs  of  obtaining  the  simulation  systems  and  supporting 
data; 

-  maximize  the  use  of  simulation  system  under  governance  of  the  project 
manager; 

-  maximize  the  acceptance  of  systems. 
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What  do  we  know  from  engineering  management? 

KNOWLEDGE  MANAGEMENT  & 
STRATEGIC  PROJECT  MANAGEMENT 
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Knowledge  Management 

•  Knowledge  management  aims  to  address  the  challenges  faced  by  modern 
organizations  of  competing  and  improving  performance  through  knowledge 

•  Knowledge  management  is  intended  to  use,  improve,  maintain,  and  create 
organizational  capabilities  to  generate  sustained  competitive  advantage  in 
organizations 

•  Knowledge  management  is  commonly  defined  as  the  processes,  tools,  and 
techniques  that  make  available  the  right  knowledge  to  the  right  knowledge 
worker,  at  the  right  time 

•  Knowledge  Management  as  the  development  of  tools,  processes,  systems, 
structures,  and  cultures  explicitly  to  improve  the  creation,  sharing,  and  use 
of  knowledge  critical  for  decision  making 

How  can  we  use  Knowledge  Management  principles 
to  support  better  (re-)use  of  M&S  for  Experimentation? 
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Strategic  Project  Management 

•  Strategic  project  management  is  the  term  lately  used  by  researchers  and 
practitioners  of  project  and  program  management  to  refer  to  the  effective 
and  efficient  management  of  project-based  organizations 

•  Strategic  project  management  focuses  on  the  best  utilization  and  alignment 
of  the  resources  of  the  project-based  organization  to  meet  its  vision  and 
goals. 

•  Strategic  project  management  focuses  on  the  providing  means  to  the 
management  enabling  them  to  provide  guidance  for  more  than  one  project 
at  a  time  in  a  way  that  the  strategic  intent  of  the  project-based  organization 
is  met 


Can  we  apply  Strategic  Project  Management 
to  enable  more  efficient  experimentation? 
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Key  Questions  to  answer? 

•  What  do  I  have  to  know  to  allow  reuse  of  M&S 

-  Assumptions  and  constraints 

-  Context  of  model  and  data 

-  Application  domain 

-  Key  entities,  processes,  and  relations 

-  Success  measures 

•  What  documentation  is  available? 

•  What  documentation  is  necessary? 

•  Are  there  any  Lessons  learned? 


How  to  we  structure  our  Operational  Analysis  Studies 
to  enable/support  respective  goals? 
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What  do  we  know  from  NATO  and  CCRP? 

NATO  CODE  OF  BEST  PRACTICE  FOR 
C2  ASSESSMENT 
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NATO  COBP  for  C2  Assessment 

•  The  NATO  COBP  for  C2  Assessment 

-  Was  developed  by  international  C2  experts  in  the 
SAS-026  activity 

-  Is  rooted  in  operations  research  methods 

-  Recommends  best  practices  for  the  structure  of  C2 
evaluation  projects 

-  has  been  adapted  as  a  standard  within  the  Joint  Staff 
and  Office  of  the  Secretary  of  Defense  (OSD) 
Networks  and  Information  Integration  (Nil) 
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NCOBP  Phases 
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Study  Hypothesis 

•  It  is  possible  to  use  the  structure  of  the  COBP 

-  To  identify  phases 

-  To  identify  required  support 

-  That  can  be  provided  by  Knowledge  Management 
and  Strategic  Project  Management 

-  Enabling  cross-domain  re-use  of  simulation 
resources. 

If  we  are  using  the  structure  of  the  NCOBP 
and  enrich  the  structure  with  KM  and  SPM  artifacts, 
Can  we  come  up  with  a  good  method? 
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What  do  we  recommend? 

PROPOSAL  FOR  A 

SYSTEMS  ENGINEERING  APPROACH 
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General  Idea 

Structure 

By  NCOBP 

/  System 
(  Engineering 
\  Process 


Repository 

By  KM 


Content 

By  SPM 
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SPM  Process  Core  Phases  and  Products 


Core  Phase 

Output  Product 

Initial  Planning 

Project  Proposal 

Refining 

Project  Work  Plan 

Implementation 

Final  Product 

Project  Review 

Lessons  Learned 
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Initial  Planning  Supporting  Documentation 


Supporting  Documentation 


Statement  of  Work  (SOW) 
Summary  of  Technical  Specifications 
_ Contractual  Constraints _ 

_ Assumptions _ 

_ Stakeholder  Analysis _ 

_ Responsibility  Matrix _ 

Work  Breakdown  Structure  (WBS) 
_ Resource  Analysis _ 

_ Cost  Analysis _ 

Study  Plan 
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Project-Based 

Organization 


Knowledge  Repository 


Project 


Sponsor’s  Problem 

*  Strategic  Importance 


Initial  Planning  Phase  (Pre-Contract) 

\ 

Problem  Formulation  (What) 

"  Statement  of  Work  (SOW) 

"  Summary  of  Technical  Specifications 
"  Contractual  Constraints 
"Assumptions 


Solution  Strategy  (How) 

"  Stakeholder  Analysis 
"  Responsibility  Matrix 
"  Work  Breakdown  Structure  (WBS) 
"  Resource  Analysis 
"  Cost  Analysis 
"  Study  Plan 


External 

Review 


Strategic  Project 
Management  Process  for 
i  Modeling  &  Simulation 
|  Federation  Development 

*  Supporting  Documents 

T.G.  Litwin  -  2008 
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Refining  Supporting  Documentation 


Supporting  Documentation 


Critical  Knowledge 
Key  Communications 
Uncertainty  Analysis 
Metrics 
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Knowledge  Repository 

-  Lessons  Learned 

-  Best  Practices 

-  Near-Miss  Events 

-  Supporting  Documents 
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Implementation  Supporting  Documentation 


Supporting  Documentation 


_ Simulation  Selection _ 

Systems  Engineering  Methodology 

Development  &  Deployment 
Data  Documentation 
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Project  Review  Supporting  Documentation 


Supporting  Documentation 


Shortfalls 
Lessons  Learned 
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Knowledge  Repository 

•  Step  One:  Capture  Documentation 

-  NCOBP  Artifacts  as  Categories 

•  Who  worked  with  Red  Cross 

•  Who  worked  in  Somalia  between  1991  and  1993 

•  Who  applied  certain  Metrics 

•  Step  Two:  Capture  Data 

-  Same  idea  as  step  one,  but  data  instead  of  text 

-  XML  with  standardized  tags 

•  Step  Three:  Ontological  Means 

-  Conceptual  representation  of  experimentation 

-  Ontology  =  Data  +  Structure  +  Processes  +  Axioms 
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What  should  you  do? 

SUMMARY  AND  RECOMMENDATIONS 
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Recommendation 

•  Project  Journals 

-  Improving  Management  within  the  project 

-  Self  awareness,  self  criticism,  and  self  development 

-  Capturing  important  events 

•  Lessons  Learned 

-  Knowledge  or  understanding  gained  by  experience 

-  Identify  common  structures  and  patterns 

•  Knowledge  Repository 

-  Text,  XML,  and  logical  structures 

-  Metadata  and  annotations 


©  2009  CCRTS 


Paper  026 


Utilizing  SPM  and  NCOBP  for  Management  of  Experimentation 


29 


Summary 

•  Recommended  approach 

-  is  well  aligned  with  current  approaches  in 

•  OA  and  C2  Assessment 

•  Strategic  project  management 

•  Knowledge  management 

-  Is  aligned  within  itself 

•  Strategic  project  management  process  (SPMP) 

•  Identifies  metadata  and  annotation  needs  for  net- 
centric  OA 

-  Is  extensible 

•  Risk  management 

•  Cost  management 
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